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In this paper, we presented two developments, the first deals with 
generalizing the modernization of the damped Dai-Liao formula in an 
optimized manner. The second development is by suggesting a monotone 
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line search formula for our new algorithms. These new algorithms with the 
new monotone line search yielded good numerical results when compared to 
the standard Dai-Liao (DL) and minimum description length (MDL) 


algorithms. Through several theorems, the new algorithms proved to have a 
faster convergence to reach the optimum point. These comparisons are 
drawn in the results section for the tools (Iter, Eval-F, Time) which are a 
comprehensive measure of the efficiency of the new algorithms with the 
basic algorithms that we used in the paper. 
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1. INTRODUCTION 
Let us define the function (1): 


F: c R” > R” (1) 
Then the issue that we discuss in this article is (2): 

F(x) =0,x EN (2) 
When the solution vector x E€ R” and the function F is continuous and satisfy the monotonous inequality i.e., 

[FŒ -FO E- y) 2 0 (3) 


There are several ways to solve this problem, such as the Newton method, and the procedure of 
conjugating gradients [1], [2]. Applications around this type of method have continued to this day [3]-[5]. Here, 
we talk about monotonous equations and their solving methods for their importance in many practical 
applications. For more details see [6]. The projection technique is one of the important methods used to find the 
solution to (1). The two researchers Solodov and Svaiter [7] gave their attention to the large-scale non-linear 
equations. Recently, many researchers implemented new articles on the topic of finding solutions to some 
constraints or unconstraint monotony equations. in various methods, so it had an aspect of interest as in [8]-[15]. 
The projection technique relies on being accelerated using a monotone case F and updating the new point 
using repetition: 
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Zk = Xk + Ady (4) 
As an initial iterative and the hyperplane is: 
Hy = {x € R” |F (zp)" (x — Zk) = 0} (5) 


To start using the projection technique, we use the update of the new point Xg+1 as given in the [6] 
to be the projection of x, onto the hyperplane Hg. So, can be evaluated (6). 


F (zx)! (xk-zk) 
X41 = PalXk — kF (Zq)] and g = EEA = (6) 


Specifically, this document is laid out: specifically, we present the suggested generalized damped 
Dai-Liao (GDDL) method in section 2. A novel monotone line search was proposed, and the penalty of 
generalized Dai-Liao (DL) parameters was determined in section 3. Section 4 proves global convergence. 
In section 5 we provide the results of our numerical experiments. 


2. GENERALIZED DAMPED DAI-LIAO 

The thinking of many researchers was concerned with finding a suitable parameter for the method 
of the conjugate gradient (CG) and imposing conditions on it to make the search direction conjugate and 
reach the smallest point of the function faster and limited steps. One of these researchers was Dai-Liao who 
provided an appropriate parameter that always makes the direction of the search in a state accompanied by a 
parameter v € (0,1), for more details see [16]. 


T T 

DL — Ik+1Yk _ j Ik+1Sk 

KS Soak V-T (7) 
dkYk dkYk 


Later Abubakar et al. [17] modified (6) by using the new formula of t and the projection technique in 
their formula. Fatemi [18] presented a generalized method for the Dai-Liao parameter and its derivation by 
applying the penalty function to the parameter to achieve a sufficient descent condition in the search direction. 
In this section we will rely on the same previous techniques with the addition of a damped quasi-Newton 
condition as in the following derivation: 


1 
q(d) = fear + Ikr1d + zd Bkyid (8) 
With Vq (ak+1dkp+1), the gradient of the model in xg+2, as an estimation of gg+2. It is easy to see that: 


Va(ar+rdk+1) = Jr+1 + C41 Braid (9) 


Unfortunately, a, in (4) is not available in the current iteration, because dg+1 is unknown. Thus, 
we modified (4) and set. 


Ir+2 = Geri tt Bgd (10) 


Where t > 0 is a suitable approximation of &g+1 . If the search direction of the CG-method. 


dk+1 = —Gui1 + Byd (11) 


In an efficient nonlinear CG method, we introduce the following optimization problem based on the 
penalty function: 


min[ iss dies +P wi l(gk+2Sk-i)” + (dk+1Yr-) II (12) 

If m = 1,2,3,4,5,..,n. Now, substituting (10), (11) in (12), and using the projection technique we obtain: 

i 2 
mn [=WFesall? + Be Gharde + PUM Sea)? + 2tF oer Sei eas” Besa Sei + t?(dp+1 Bus1Sk-i) + 


(Fht1Yk-i)? — 2bkFk+1Yr-idkYk-i + (Beaty)? (13) 
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After some algebraic abbreviations, we get the: 


1 
Pk = F [—Frzad + 2P Xiti FhrYk-idkYr-i + 2t°P Dika Fig Bis 1 Seid Besa Sk-i — 
2tP yey Fe+1Sk-i dk Bes 1Sk—i] (14) 


Where: 
p= 2t?P Xit 2 (dF Bk+1Sk- D? +2P X a (aE Yk-i)? 


To get a new parameter, let us assume that the Hessian approximation B,., to satisfy the extended 
damped quasi-Newton (QN) equation and with the incorporation of the use of projection technology we get: 


1 1 Z 
Bk+1Sk-i = g; TH ki + (1 — 1) By Si) = ngei = Ve-i (15) 
And é; is the projection step. 


. T T 
1 İf Sk-iYr-i Z Sk-iBkSk-i 


tT = nsl_;BkSk-i Í (16) 
q i if Sk-iYr-i < Sk-iBkSk-i 


T T 
Sk—iBkSk-i™Sk-iYk-i 


Then, 
new — -Firik + Lie 1 Pyk- idkYk-i ae Fah hk i 
A DIAA a Je a +(dk yk- i) ^ m (eag Th y +(d7 yk- i) *) m (eag TR. D +(d7 yk- i) 3) 
tHE 1Frask- A i (17) 


(eaga) +(e) ) 


So, there are two possible scenarios for this parameter such that, case I: if sf_iVx—-i = Sh—;BxSx-; then Tp = 1 


and F2; = o 


tom „T T 
T eo hint Fk+1Yk-iâkYk-i 
newl — Frei dk ie 1 Fraye- id} Yk- i Ep 


+ 
5 2 
20 y (lafy) “(ahd ) i= m(n i) +(dk Yk- i) l i= (Saba) i) + (ap Yk- i) i 


tom pT T. 
Fy i=1Fk+1Sk-ilkYk-i 


a3? (18) 
i m(n) i) +(d] yk- D ‘) 
Put Sk = Edy. 
m T T T 
new1 -2i akae- dhe i Skt Lind Feta Skid Vk-i SkFe445k 19 
n = = (19a) 


2 2 
m (afyk- D 2+1) m (dpyr-i) 2P1(t2+1) X? (dfyp-i) 


To investigate the proposed method when P} approaches infinity, because by making this coefficient 
larger, we penalize the conjugacy condition and the orthogonality property violations more severely, thereby 
forcing the minimizer of (10) closer to that of the linear CG method. 


T T T T 
newt — Xica Fk+1Yk-idkYk-i Ext Dy Fea Skid Veil (19b) 
k = T 2 2 T: A 
Vi (dkYk-i) 2+1) ym (dfyp-i) 


When compared to (6), we notice that the value is: 


ur = v € (0,5) (20) 


(t2+41) ’ 


D 
Case I: if sf_iYg-i < Sk-iBpSpg-i then Ypi = a 
k 


from (12) and Sk = čęķdę by projection, the technique to 


convert the pge” form to: 
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prew2 = | ire 1 Fryk- idk Vk-i trie 1 Fersk- idkYki |+ E tie 1 Fyk- idkYk-i = 
Tease yk- D +R (GLY K- i) *) rm, (t2 (a7 yp- D +42 (aL yk- D) re t2(a] yP X +42 (dT yp i) *) 
Šk FeyiSk | (21) 
2p, 5M,(t2(akyP_;) +8R(ak vei) ) 


Now using algebraic simplifications, we obtain: 


T 
new2 _ 1 m T T m 4&kSk-i ym T T 
Pk = A (Dealt FEY = toh Seal + Xica dyka ierlt3Fk+1Yr-i = t4Fk+1Sk-il — 
=p 


sains) (22a) 
i.e., 

p° = Dit? (dive? + Fe (GE ed") 

tı = t?t + &? and t, = t &t, — t?t}(1 — Tg) 

t3 = t°Tk(1 — Tp) and t, = t & (1 — Te) — t° (1 — t,)*) 


As we talked about (P,) then (P) when you come close to infinity, then we use the parameter 
omitted from this limit: 


2 1 di sp; 
Pk” = A (Dales FEY z t2Ff+1Sk-il Nici EPR ealta Fk Y-i = tal Bs5e-i])(22b) 


To obtain better results as in section 5. We have another paper on this type of method, but without 
generalizing the original formula [17]. 


3. NEW PENALTY PARAMETER 

Due to the importance of concomitant gradient methods in this paper, we highlight them in the 
derivation of new algorithms. Now, we will derive more coefficients of the penalty function. Although, in this 
formula, we will focus on the two new parameters defined in the (19) and (22) respectively, then we will check 
and update the derived parameters by relying on the appropriate direction of regression of the CG method: 


3.1. Lemma 1 
Assume that the sequence of the solution generated by the method (19) with monotone line search, 
then for a few positive scalars 6, and ô, satisfying 6; + 62 < 1, we have: 


Flridk+1 S = -(1 ~ by i ô2)||Fk+1ll? (23) 
When: 
2 52(yp sx) 
it — 1] < Tse 572 lisp 2H) 
P, = 2 ô1llFk+1ll? <A <1 (25) 


m(t?+1) n mox x (On- i—0.5 AiSk- OT Frei)” 
Proof: if we used (5) and (14) then: 


1 


Fidri = —MFesill? + aD 


z CO (Firs O s9FEuso) - 


LEE isk) 


ke (uas) (26) 


2 
2Ps(744) 3, (vf isk) 


Since A, < 1, implies that: 
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SG O : oT L T 
m i= - = 
Feriden SWF? + Ee 1( Ore-i — 0-5; Si)” Fess) OF iS) F180) + 
i=1Wx-i5 
1_ Ske E (lease). 
E as) 5" Of sw wie Fes Ski) 1k-iSk) Fes Sk) — 2P, Piece ay (27) 


: ne ; t 1 23 
Using this inequality xy < ae + a where x, y and t’ are positive scalars, we have: 


t! 
Fiyidk+1 < —|llFk+ ll + DEK 1(Qr-i — 0.54iSk-i)" Fea)? Ok-iSk)? + 
i=1Wk-i 
m T Dons Šk T 2 Grt-D)? om or T 
DE (Esse) t Ersisi) 2P, 5 (9i ise) (24) (Fie+aSi) t 2y sk(t?+1) imilSe—iFic+al Fie Se (28) 


Let t’ = 2mP,(t? +1), 


Pym(t?4+1 
Fidesi < -lger ll? +272 max (Ori = 0.5451)" Fess)? + 
(Ert-1)? 
Diatr Vinal Skil [Fess Se (29) 


By Cauchy-Schwarz inequality implies: 


P4(t?+1 
Fladen <- [1 -ZEEE max (Ori = 0585-0" Fer)? - 
(Ext-1)? 
Pes lll Balsall Fel (30) 
Frida. < —( — 6; — 62) < —p1]lFk+1 ll? (31) 


Since t is an approximation of the step size, we use the updated formula: 


2 82(ypsx) 
Se if Wet — US ae sca 
2 S2(yve Sk) 
1 [_2 Saves) _ O.W. 
+ ise D7 iseal 


Hence, the proof is completed. 


o 
ll 


(32) 


3.2. Lemma 2 
Assume that the solution sequence is generated by the new method (16) with a monotone line 
search, then for a few positive scalars 63,64, 65 and 6, satisfying 63 + 6, + ôs + dg < 1, we have: 


Fividnas S = =(1 Py 63 E ô4 i ôs z 56) Fits ll? (33) 


2 Es 
ltz — 2) < [ZOO g jt, — 2) < y2 8t T) (34) 
sell X;+4llSx-ill 


And 
2 ô3|lFk+1ll? 2 ôs|lFk+1ll? 
Pyq = 3 dP,, = 
2a m max (Cta VR-i-0.5 Aise-DT Frei) ‘ce a m pmax (Cts Yk-i—0.5 Ase) Fk) GD) 
Ai < 1 is a scalar. 
Proof: we substituting (16) in (9) and multiplying by Fķ+1 that: 
1 
Firidk+ = —ÌlFk+ ll? + 4 realty Fee Ved Yk-iSk) (Fersk) — te Fier Ski) Or-iSk)(Fk+1Sk)] + 


al. l 
Die a alta Fkt1Yk-i) V k-iSk)(Fk+1Sk) — EES- DOES Fhe 50)]) = wd (Fit 15x)? (36) 
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1 1 f1 
Fiza. = Z llFg+ ll? + pa i= (Yki — O.5A;S¢—i)" Feat) Ok-iSk)(Fk+1Sk) + A 
1 
to | Ea Fiers) Okas Fa Si) + EECC aY — 0-548)” Fees) (SE Se) Fea Sie) + 
1 fi Š 
aa tal Dis e Ea — spa Fitea Si)? (37) 


By following the same steps in lemma 3.1 we get: 


(t2-2)? 
Feeds. S Z |llFesill? + Pza Diel lti Yk-i — 0.5A; Sei)" Fg+1)? + IA ealse—iFcsal IFES] + 
(t4-2)? 
Pop Dini ((t3Vx-i — 0.54iSk-i) Fkt)? + or Lea lst—iFeas lFic+1 Sl (38) 


2t? (1-1k)? 54 SK 


By Cauchy-Schwarz inequality implies: 


(t2-2)? 
1—mP2, max (Vai — 0.5; Spi)? — or lise ll ia IlSx—all — 


FR dpa S — Fics ll? (39) 


2 (t4-2)? 
mPzp MaX (taYr-i — 0.54iSpg-i)" — aT lisk ll Xz llsk-:ll 


Then, Fi dias < —(1 — 63 — 64 — ôs — 56) [IFrasll? < —p2llFk+1 ll? 


The proof is completed. In the next paragraph we talk about the Algorithm 1 (minimum description length 
conjugate gradient (MDL-CG)) used for numerical comparison which is an update of the Dai-Liao algorithm: 


Algorithm 1. MDL-CG [19] 


Given xo E 0,1r,0 E (0,1), stop test € > 0, set k = 0. 
Step 1: evaluate F (xx) and test if ||F (x;,)|| < € stop else goes to step 2. 


Step 2: evaluate d, by (9) and substitute BY?" = Fibs Yk L v Fits and vz = p Well? _ she gq, = 0 
ý k vesk WSK KO E skye MisE O 
is the stopping criterion. 
Step 3: compute Zk = x, + a,d,, with the step-size a, = r™”k . 
=F (xp +r” dy)" dy > or™||F (Œp + 7d, MMe ll? (40) 


Step 4: check if z, € Q and ||F(Z,)|| < € stop. 
Step 5: let k = k + 1 and go to step 1. 


The new Algorithm 2 (GDDL-CG) is a generalization according to the value of m and it is an update and 
suppression of the Dai- Laio algorithm as in the steps: 


Algorithm 2. GDDL-CG 


Given xo E 0,r,0,4,y E (0,1), stop test € > 0, set k = 0. 

Step 1: evaluate F (xx) and test if ||F (x;,)|| < € stop else goes to step 2. 

Step 2: when yj Sk 2 Sf Sg compute P} from (25) and if P, # œ then 67°" from (19a) else (19b). 
Step 3: when yj Sk < Sf Sg compute P, from (35) and if P, # œ then £7”? from (22a) else (22b). 
Step 4: compute dx by (9) and stop if dẹ = 0. 

Step 5: set Zk = Xk + gdp, where a, = r™ with m to be the shortest positive number m so: 


d 2 
-F (xp + yr dy)"dy > oyr™ [ulldy|l? + (1 — w BET] (41) 


Step 6: if Zg E€ Q and ||F(z,)|| < € stop, else compute the point x,4, from (6). 
Step 7: let k = k + 1 and go to step 1. 


4. GLOBAL CONVERGENCE 

In the previous section, we gave a preface to the proof of convergence condition by establishing the 
property of sufficient descent through lemmas 3.1 and 3.2. In the beginning, there are a set of assumptions 
that we mention in this section, and then we move on to theories. Adding several lemmas for the step length 
of the new algorithm. Now we need some assumption, to begin with, the proof of convergence condition, 
which is illustrated thus: 
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4.1. Assumption 

Suppose F fulfills the following assumptions: 
a) The solution group of (2) is non-empty. 
b) The function F is Lipschitz continuous, i.e., 


F(x) — FQ) || < Lilx — yll, Y x,y € R” (42) 
c) F satisfies, 
(F(x) — F (y), x — y) = ellx — yll? ,Y x,y € R”, c>0 (43) 


4.2. Lemma 1 
Assume (x € R”) satisfy F(x)=0 and {x} is generated by the new algorithm GDDL-CG. 
If lemmas 3.1 and 3.2, hold, then ||x;,4, — XI|? < Ilx — XII? — |lxn41 — xgl|? , and 


Èr=ollXk+1 — Xgll < (44) 


4.3. Lemma 2 
Suppose {x} is generated by the new algorithm GDDL-CG then: 


Proof: since the sequence {||x, — X||} is not increasing; {xx} is bounded, and lim llxz+1 — Xz || = 0. From (3) 
00 


using a line search, we have: 


|F(zK)? (x-zx)| |e F (zx)* dk] 
ire IIF (z)Il = Fe = > axl|ldxl| = 0 (46) 


llxk+1 — Xxll = 
Then the proof is completed. 


4.4. Theorem 
Let {xx} and {Zg} be the sequences generated by the new algorithm GDDL-CG then: 


lim inf ||F(x;,)|l = 0 (47) 
k- 00 
Proof: case I: if lim inf | \|d, || = 0, then lim inf IIF (xx) || = 0. The sequence {xg} has some accumulation 
k> 
point x such that Fo) = 0. Hence, {||x, all} converges to x. Case II: if lim ay, \|d,|| > 0, then 
lim inf ||F(x,)|| > 0. Hence lim a, = 0. Using the line search. 
k>0 = 
m T m 2 lldrll? 
-F (xy + yrr dy)" dy > oyr™ [ulldy|l? + (a — w BAG] (48) 
And the boundedness of {xp}, {dp}, yields 
—F(x)'d < 0 (49) 
From (17) and (23) we get: 
—F(x)"d = p; IE®IP > 0 (50) 


The (31) and (32) indicates a contradiction for i = 1,2. So, lim inf ||F(x,)|| > 0 does not hold and the 
k-0o 


proof is complete. 


5. NUMERICAL PERFORMANCE 

In this section, we present our numerical results that explain the importance of the new algorithm 
GDDL-CG compared to the MDL-CG algorithm [19] using Matlab R2018b program in a laptop calculator 
with its Core™i5 specifications. The program finds the results on several non-derivative functions through 
several two initial points. In Table 1, we review the details of the initial points used to compare algorithms as 
shown in Table 1. 
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Table 1. Number of initial points 


Name of variable Number 
X4 (1,1,1,..,1)7 
Xz (rand, rand, rand,.., rand)" 


Table 2. Information of test functions [20]-[26] 


Name of functions Details Reference 
F F;(x) = 2 x; — sin|x;| [21] 
F, F(x) = x; — sin(x;) [21] 
F, F(x) = e%i-1 [22] 
Fa F,(x) = Ve(x — 1), for i = 2,3,..,n— 1 [22] 
n 


E _1 2 1 
ORSA “| 
j=1 


forc=1*10° 


bat 
fs Fe) = malt D- [z9] 
Fe F,(x) = min(min(|x;|, x7) ,max(|x;|, xř)) [23] 
cos(x1+X2) 
ái RO) =x e mT a 
cos(xj44+X{+Xj_-1) , 
F(x) =x- e n+I , for i= 2,3,..,n— 1 
COS(Xn-1+Xn) 
F(x) =x,-e nt 
i 
Fe F(x) = ae -1 21] 
Fy F,(x) = e™1 — 1, F;(x) = e* — xi — 1 20] 
Fio te 25] 
Ra) =) al 
izi 
Fii Z 25] 
F(x) = È lal 
izi 
Fi, F(x) = max |x;| 25] 
i=1,..n 
Fy3 Z PSR 25] 
FG) = Y [x] e7 žhe 
izi 
Fig = 26] 
F(x) = Y J sinfi) + 0.1 Œ) 
izi 
Fis 7 ; 26] 
RG) = > a 
i=1 


The n-dimensional versions of these techniques are implemented here (1000, 2000, 5000, 7000, 
12000). The stopping criterion is ||F(x;,)|| < 1078. All algorithms of this kind may be distinguished from 
one another based on their performance in terms of (Iter): the number of iterations, (Eval-F): the number of 
evaluations of functions, (Time): in seconds measured by the CPU, and (Norm): the approximate solution 
norm. In Table 2, we mention the details of the test problems F(x) = (fi, fo, fa» ---» fy)’ used with the 
references from which they were taken, the points x = (x1, %2,X3,..., Xn)’, and Q = RÌ. 


wy 
08} ¢ -$ ost $it r 
H $ $i ; : 
i H tit $ : 
07t $ F ? o7+ 323 i : 
H $ $ 7; i 
$ r $ a os 
§ 08) ii f H 5 06 : t oł 
B $ Ẹ ; ; 
S $i £ $ e l 
$08 i H 205 ? 3 i E 
g H ? > noi, e; 
Ò 04 H v s. . $ 
E ji ; € 04 tle 3 
§ oo ee | 
£ pE f g T DE SE $ 
O 5 f 
ae = H +—DL-CG $03 $$ DL-CG 
g H *—MDL-CG F] tt +—MDL-CG 
o2) i *—GRA-DLm=5 a +t *— GRA-DL m=5 
H + GRA-DLm=4 0.2 ce A +— GRA-DL m=4 
o4 i = GRA-DLm=3 | | ? + — GRA-DLM=3 
H *— GRA-DLm=2 01 t +— GRA-DL m=2 
~ GRA-DLm=1 $ © GRA-DLm=1 
ot = : 
0 5 10 15 20 25 ya 10 15 
( 5 
Tau Tau 
(a) (b) 


Figure 1. Performance of the seven algorithms with respect to (Iter): (a) about x, and (b) about x, 
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Using style for figures as in [19], the following three figures are for comparison between the old DL 
and MDL algorithms with the new algorithm when switching the value of the generalization i.e., from 1 to 5 to 
get 5 new algorithms. New algorithms are considered generalizations and can generalize to more than 5. 
To accurately know the impact of algorithms, we drew the following figures. 

Figure 1 shows the effect of the new algorithms when taking the iterations tool as a measure of the effect, 
and the figure was divided into two parts, i.e., Figure 1(a) when calculating the point x,, and Figure 1(b) when 
calculating the point x2. As for Figure 2, it shows the effect of the new algorithms when taking the function 
number calculation tool as a measure of the effect, and the figure was divided into two parts, i.e., Figure 2(a) when 
calculating the point x,, and Figure 2(b) when calculating the point x. Finally, Figure 3 shows the effect of the 
new algorithms when taking the time spent in calculations tool as a measure of impact, i.e., Figure 3(a) when 
calculating the point x,, and Figure 3(b) when calculating the point x2. 


tees 


teteseeeeesss 4 


0.4} ——DL-CG 
—+—MDL-CG 

0.3 f —+—GRA-DL m=5 

GRA-DL m=4 

02+ — = GRA-DL m=3 | 5 

GRA-DL m=2 

0.1} = GRA-DL m=1 0.1 


Performance of function evaluation 
o 
uo 

Performance of function evaluation 
Oo 
a 
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o 
a 
— |oocoooooposocooooogocoosoocogpee 
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Figure 3. Performing the seven algorithms with respect to (Time): (a) about x; and (b) about x2 


Through the use of the three figures, we can conclude that the new algorithms are superior to the 
two algorithms with which we compared our work. Furthermore, we have discovered that the random starting 
point with which we began our work brought to light the significance of the new algorithm when 
generalizing m = 2 when calculating the number of iterations and was the best of the new algorithms when 
calculating the number of times, the function is calculated. The algorithm with the value of m equal to one 
performed the best and was superior to the others. In conclusion, when considering the amount of effort 
invested in developing these algorithms, the two algorithms with m equal to 5 performed the best. 
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6. CONCLUSION 

Our numerical results in the previous section, represented by three numbers, show the efficiency of 
the generalized algorithm GDDL when compared with the previous two algorithms. Its efficiency is better 
when we take the randomly generated point as a starting point and this will increase the efficiency when the 
dimensions used in the variables increase, which shows greater stability and makes the new generalized 
algorithm more suitable than other previous algorithms for the existence of the limit containing the penalty 
parameter in the new generalized algorithm. Theoretical proofs of the new algorithm give the proposed 
algorithms more power than the previous algorithms. This is why these algorithms are considered successful 
in both theoretical and numerical aspects. 
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